System and method for multiple node display

ABSTRACT

A method of display of visual content is disclosed which utilizes nodes that launch only when necessary. The nodes may be rendered and/or displayed in the coordinate system of another node, perhaps a parent node, when appropriate. Increase precision of display objects is achieved.

RELATED APPLICATIONS

This application claims the benefit of provisional application Ser. No.60/474,313, the disclosure of which is hereby incorporated by referencein its entirely.

BACKGROUND OF THE INVENTION

The present invention relates to zooming user interfaces (ZUI) forcomputers.

Most present day graphical computer user interfaces are designed usingvisual components of a fixed spacial scale. The visual content can bemanipulated by zooming in or out or otherwise navigating through it.However, the precision with which coordinates of various objects can berepresented is extremely limited by the number of bits, usually between16 and 64, designated to represent such coordinates. Because of theirlimited representational size, there is limited precision.

In the context of the zooming user interface, the user is easily able tozoom in, causing the area which previously covered only a single pixelto fill the entire display. Conversely, the user may zoom out, causingthe contents of the entire display to shrink to the size of a singlepixel. Since each zoom in or out may multiply or divide the xycoordinates by numerous orders of magnitude, just a few such zoomscompletely exhaust the precision available with a 64 bit floating pointnumber, for example. Thereafter, round-off causes noticeable degradationof image quality.

It is an object of the present invention to provide a ZUI in which alarger range of zooms is possible.

It is a further object of the invention to provide a ZUI in which theprecision in which coordinates are represented is related to therequired precision needed at a particular zoom level of detail. It is afurther object of the present invention to allow a pannable and zoomabletwo-dimensional space of a finite physical size, but of an arbitrarilyhigh complexity or resolution, to be embedded into a well-defined areaof a larger pannable and zoomable two-dimensional space.

A further objective of the present invention is to allow zooming outafter a deep zoom-in to behave like the “back” button of a web browser,letting the user retrace his or her steps through a visual navigation.

A further objective of the present invention is to allow zooming inimmediately after zooming out to behave analogously to the “forward”button of a web browser, letting the user precisely undo the effects ofan arbitrarily long zoom-out.

A further objective of the present invention is to allow a node, avisual object as defined more precisely below, to have a very largenumber of child nodes (for example, up to 10{circumflex over ( )}28).

A further objective of the present invention is to allow a node togenerate its own children programmatically on the fly, enabling contentto be defined, created or modified dynamically during navigation.

A further objective of the present invention is to enable near-immediateviewing of arbitrarily complex visual content, even if this content isultimately represented using a very large amount of data, and even ifthe data are stored at a remote location and shared over a low-bandwidthnetwork.

A further objective of the present invention is to allow the user tozoom arbitrarily far in on visual content while maintaining interactiveframe rates.

A further objective of the present invention is to allow the user tozoom arbitrarily far out to get an overview of complex visual content,in the process both preserving the overall appearance of the content andmaintaining interactive frame rates.

These and other broader objectives of the present invention will becomeapparent to those skilled in the art from a review of the specificationthat follows.

SUMMARY OF THE INVENTION

The above and other objects of the present invention are accomplished bydisplaying visual content as plural “nodes.” Each node preferably hasits own coordinate system and rendering method, but may be containedwithin a parent node, and may be represented in the coordinate systemand rendering method of the parent node. As a user navigates the visualcontent, by for example, zooming in or out, a node is only “launched”when the zooming results in an appropriate level of detail. Thelaunching of the node causes the node to be represented in its owncoordinate system and/or rendering method, rather than in the coordinatesystem and/or rendering method of a different node.

Prior to the node being launched, the node is either represented in thecoordinate system of the parent node, or not represented at all. Bylaunching nodes only when they are required, the precision of acoordinate system is a function of the zoom level of detail of what isbeing displayed. This allows a variable level of precision, up to andincluding the maximum permissable by the memory of the computer in whichthe system operates.

DESCRIPTION OF THE DRAWINGS

For the purposes of illustration, there are forms shown in the drawingsthat are presently preferred, it being understood, however, that theinvention is not limited to the precise arrangements andinstrumentalities shown.

FIG. 1 is a depiction of visual content on a display;

FIG. 2 is an image of the visual content of FIG. 1 at a different levelof detail;

FIG. 3 is a representation of an embodiment of the invention;

FIG. 4 is an exemplary embodiment of the invention showing plural nodeson a display;

FIG. 5 is a tree diagram corresponding to the exemplary embodiment shownin FIG. 4.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

We assume a user interface metaphor in which the display is a camera,through which the user can view part of a two-dimensional surface, or 2Duniverse. For convenience, although it is not necessary to do so, weascribe physical dimensions to this universe, so that it may be, forexample, one meter square. The invention is equally applicable toN-dimensional representations.

The exemplary universe in turn contains 2D objects, or nodes, which havea visual representation, and may also be dynamic or interactive (i.e.video clips, applications, editable text documents, CAD drawings, orstill images). For a node to be visible it must be associated with arendering method, which is able to draw it in whole or in part on somearea of the display. Each node is also endowed with a local coordinatesystem of finite precision. For illustrative purposes, we assume a nodeis rectangular and represented by a local coordinate system.

These two parameters, the rendering method and coordinate system,specify how to display the node, and the positions of items in the node.Each node may have 0 or more child nodes, which are addressed byreference. The node need not, and generally does not, contain all theinformation of each child node, but instead only an address providinginformation necessary to obtain the child node. As a user navigates, forexample, zooms in and out, the nodes are displayed on the screen, asshown, for example in FIG. 1.

Generally, a “node” is the basic unit of functionality in the presentinvention. Most nodes manifest visually on the user's display duringnavigation, and some nodes may also be animated and/or respond to userinput. Nodes are hierarchical, in that a node may contain child nodes.The containing node is then called a parent node. When a parent nodecontains a child node, the child's visual manifestation is alsocontained within the parent's visual manifestation. Each node has alogical coordinate system, such that the entire extent of the node iscontained within an exemplary rectangle defined in this logicalcoordinate system; e.g. a node may define a logical coordinate systemsuch that it is contained in the rectangle (0,0)-(100,100).

Each node may have the following data defining its properties:

-   -   the node's logical coordinate system, including its logical size        (100×100 in the above example);    -   the identities, positions and sizes of any child nodes,        specified in the (parent) node's logical coordinates;    -   optionally, any necessary user data;    -   executable code defining these operations or “methods”:    -   initialization of the node's data based on “construction        arguments”    -   rendering all or a portion of the node's visual appearance (the        output of this method is a rendered tile);    -   optionally, responding to user input, such as keyboard or mouse        events.

The executable code defines a “node class”, and may be shared among many“node instances”. Node instances differ in their data content. Hence anode class might define the logic needed to render a JPEG image. The“construction arguments” given to the initialization code would theninclude the URL of the JPEG image to display. A node displaying aparticular image would be an instance of the JPEG node class. Pluralinstances of a node may be viewable in the same visual content, similarto the way a software application may be instantiated numerous timessimultaneously.

Note that in a complex visual document or application, it is usuallypossible to divide the necessary functionality into nodes in manydifferent ways. For example, a scripted web-page-like documentcontaining multiple images, pull-down menus and buttons could beimplemented as a single node with complex rendering and user inputmethods. Alternatively, it could be implemented as a parent node whichonly defines the overall layout of the page, with every constituentimage and button a child node. This has the obvious advantage of reusingor “factoring” the functionality more effectively: the buttons may allhave the same behavior, and hence all be instances of the same nodeclass; the images may all be in the same format and so also be instancesof a common node class, etc. This also simplifies rearranging thelayout—the parent node can easily move or resize the child nodes.

In accordance with the present invention, visual content may bedisplayed in a manner that depends upon the state of navigation input bya user. For example, FIG. 1 shows a node 105 which may be the image of aportion of the city. Node 105 may contain child nodes 101-103. Node 101may be an image of a building in the city, node 102 could be an image ofa playground, and node 103 might be a sports arena. At the level of zoomshown, nodes 101-103 are relatively small, so they can be represented asa small darkened area with no detail in node 105, located at the correctlocation in the coordinate system of node 105. Only the coordinatesystem and the rendering method of node 105 is needed.

Consider the case where the user now zooms in so that a different levelof detail (LOD) such as that shown in FIG. 2 results. In the LOD of FIG.2, nodes 101 and 102 are no longer visible on the screen, due to thefact that the visual content is displayed as much larger. Additionally,it is noted that the because the size at which sports arena node 103 isdisplayed is now much larger, the details of the sports arena, such asthe individual seats, the field, etc, now must be displayed.

In furtherance of the foregoing, sports arena node 103 would now bedisplayed not as a darkened area with no detail in the coordinate systemof node 105, but rather, it would be “launched” to be displayed usingits own coordinate system and rendering method. When displayed using itsown coordinate system and rendering method, the details such as seating,the filed of play, etc. would be individually shown. Other functionsdiscussed above, and associated with the node 103, would also beginexecuting at the point when node 103 is launched. The particularnavigation condition that causes the launching of node 103, or any nodefor that matter, is a function of design choice and is not critical tothe present invention.

The precision with which the node 103 will be displayed is the combinedprecision of the coordinate system utilized by node 105, as well as thatof node 103. Thus, for example, if the coordinate system of each of saidnodes utilizes 8 bits, then the combined precision will be 16 bitsbecause the coordinate system of node 103 is only utilized to specifythe position of items in node 103, but the overall location of node 103within node 105 is specified within the coordinate system of node 105.Note that this nesting may continue repeatedly if sports arena 103itself contains additional nodes within it. For example, one such node201 may in fact be a particular concession stand within the sportsarena. It is represented without much detail in the coordinate systemand rendering method of node 103. As a user continues zooming in onsports arena 103, at some point node 201 will launch. If it is displayedusing 8 bits of precision, those 8 bits will specify where within thenode 201 coordinate system particular items are to be displayed. Yet,the location of node 201 within node 103 will be maintained to 8 bits ofprecision within the coordinate system of node 103, the location ofwhich will in turn be maintained within the coordinate system of node105 using 8 bits. Hence, items within node 201 will ultimately bedisplayed using 24 bits of precision.

By nesting nodes within nodes, the precision at which visual content mayultimately be displayed is limited only by the memory capacity of thecomputer. The ultimate precision with which visual content in a node isdisplayed after that node is launched is effectively the combinedprecision of all parent nodes and the precision of the node that haslaunched. Hence, depending upon the level of nesting, the precision mayincrease as needed limited only by the storage capacity of the computer,which is almost always much more than sufficient. Additionally, theincreased precision is only utilized when necessary, because if theimage is at an LOD that does not require launching, then in accordancewith the above description, it will only be displayed with the precisionof the node within which it is contained if that node has been launched.Thus, for nodes nested within other nodes, as one moves from theoutermost node inward, one may traverse nodes that have launched untilfinally reaching a node that has not launched yet. Any such unlaunchednode, and nodes further within it, will be displayed only with theprecision of the last traversed node that has launched.

This results in an “accordion” type precision, wherein the precision atwhich visual content is displayed expands and contracts as necessary andas dictated by the navigational input of the user, maximizing theefficiency of system resources by using them only when necessary forhigher precision.

It is also noted, that when a node launches the display of that nodechanges from being based upon the coordinates and rendering method ofthe parent node to the coordinates and rendering method of the childnode. That change is optimally made gradual through the use of blending,as described, for example, in copending U.S. patent application Ser. No.10/790,253. However, other methodologies of gradually changing from thedisplay of the information in the coordinate system and rendering methodthe parent node to the child node are possible. The system could beprogrammed, for example, that over a particular range, the blending fromparent to child occurs. Then, as the user traverses through that rangeduring a zoom, the changeover occurs, unless the navigation is ceasedduring that range, in which case the blending may continue until fullydisplayed in the appropriate coordinate system.

An additional issue solved by the present invention relates to a systemfor maintaining the spatial interrelationship among all nodes duringdisplay. More particularly, during dynamic navigation such as zoomingand panning, many different coordinate systems are being used to displaypotentially different nodes. Some nodes, as explained above, are beingdisplayed merely as an image in the coordinate system of other nodes,and some are being displayed in their own coordinate systems. Indeed,the entire visual display may be populated with nodes displayed atdifferent positions in different coordinate systems, and the coordinatesystems and precisions used for the various nodes may vary duringnavigation as nodes are launched. Hence, it is important to ensure thatthe nodes are properly located with respect to each other, because eachnode is only knowledgeable of its own coordinate system. The presentinvention provides a technique for propagating relative locationinformation among all of the nodes and for updating that informationwhen needed so that each node will “know” the proper position in theoverall view at which it should render itself.

The foregoing may be accomplished with the addition of a field to thenode structure and an additional address stack data structure. Theexpanded node definition includes a field which we term the “view”field, and which is used by the node to locate itself relative to theentire display. The view field represents, in the coordinates of thatnode, the visible area of the node—that is, the image of the displayrectangle in the node's coordinates. This rectangle may only partiallyoverlap the node's area, as when the node is partially off-screen.Clearly the view field cannot always be kept updated for every node, aswe cannot necessarily traverse the entire directed graph of nodes inreal time as navigation occurs.

The stack structure is defined thus:

-   -   Stack<Address>viewstack;    -   where this stack is a global variable of the client (the        computer connected to the display). For exemplary purposes we        assume that navigation begins with an overview of a universe of        content, defined by a root node; then this root node is pushed        onto the viewStack, and the root node's view field might be        initialized to be the entire area of the root node, i.e.    -   rootNode.view=rootNode.coordSystem;    -   Push(viewStack, rootNode);

Schematically, the viewStack will specify the addresses of a sequence ofnodes “pierced” by a point relative to the display, which we will takein our exemplary implementation to be the center of the display. Thissequence must begin with the root node, but may be infinite, andtherefore must be truncated. In an exemplary embodiment, the sequence istruncated when the nodes “pierced” become smaller than some minimumsize, defined as minimumArea. The current view is then represented bythe view fields of all of the nodes in the viewStack, each of whichspecify the current view in terms of the node's local coordinate system.If a user has zoomed very deeply into a universe, then the detailedlocation of the display will be given most precisely by the view fieldof the last node in the stack. The last element's view field does not,however, specify the user's viewpoint relative to the entire universe,but only relative to its local coordinates. The view field of the rootnode, on the other hand, does specify where in the universe the user islooking. Nodes closer to the “fine end” of the viewStack thus specifythe view position with increasing precision, but relative toprogressively smaller areas in the universe. This is shown conceptuallyin FIG. 3, where it can be seen that of the three nodes that have beenlaunched, node 303 provides the most accurate indication of where theuser is looking, since its coordinate system is the “finest”, but node301 provides information, albeit not as fine, on a much larger area ofthe visual content.

The problem then reduces to the following: the views (i.e. view fields)of all visible nodes must be kept synchronized as the user navigatesthrough the universe, panning and zooming. Failure to keep themsynchronized would result in the appearance of nodes moving on thedisplay independently of each other, rather than behaving as a cohesiveand physically consistent 2D surface.

Changing the view during any navigation operation proceeds as follows.Because the last node in the viewStack has the most preciserepresentation of the view, the first step is to alter the view field ofthis last node; this altered view is taken to be the correct new view,and any other visible nodes must follow along. The second step is topropagate the new view “upward” toward the root node, which entailsmaking progressively smaller and smaller changes to the view fields ofnodes earlier in the stack. If the user is deeply zoomed, then at somepoint in the upward propagation the alteration to the view may be sosmall that it ceases to be accurately representable; upward propagationstops at this node. At each stage of the upward propagation, the changeis also propagated downward to other visible nodes. Hence, first, thelast node's parent's view is modified; then, in the downwardpropagation, the last node's “siblings”. The next upward propagationmodified the grandparent's view, and the second downward propagationmodifies first uncles, then first cousins. The downward propagation ishalted, as before, when the areas of “cousin nodes” become smaller thanminimumArea, or when a node falls entirely offscreen.

The foregoing technique involves translating the layout of the variousnodes into a tree, which conceptually is illustrated in FIGS. 4 and 5.As can be seen from FIGS. 4 and 5, there is a corresponding tree for aparticular displayed set of nodes, and the tree structure may be used topropagate the view information as previously described.

A panning operation may move the last node far enough away that it nolonger belongs in the viewStack. Alternatively, zooming in might enlargea child to the extent that a lengthening of the viewStack is required,or zooming out might bring the last node's area below a minimum arearequiring a truncation of the viewStack. In all of these cases theidentity of the last node changes. These situations are detected duringthe downward propagation, which may alter the viewStack accordingly,potentially leaving it longer or shorter.

One simple case of the foregoing is that during zooming, a node getslaunched so that now it needs to be placed in the view stack. Anotherexample is that by zooming out, a previously visible node becomes sosmall that it must be removed from the viewstack.

An extension of the idea is to avoid truncating the viewStackimmediately in response to a long outward zoom. Truncating the viewStackis only necessary if the user then pans. Although a long outward zoomwill cause the view fields of deeply zoomed nodes to grow very large(and therefore numerically inaccurate), a field

-   -   Point2D viewcenter;    -   can be added to the Node structure, representing the central        point of the view rectangle; zooming without panning therefore        does not alter the viewCenter field of any node. This        construction allows zooming far outward to be followed        immediately by zooming back in. Because the viewStack has been        left intact, the user can then return to precisely the starting        view. This behavior is analogous to the “back” and “forward”        buttons of a web browser: “back” is analogous to zooming out,        and “forward” is analogous to zooming back in. In a web browser,        if a user uses “back” to return to a previous web page, but then        follows an alternate link, it is at this point that “forward”        ceases to work. Following an alternate link is thus analogous to        panning after zooming out.

The foregoing provides that visual content may be displayed andnavigated in a variety of fashions with substantially infiniteprecision, limited only by the capacity of the computer system on whichthe application is running. The visual content displayed at any giventime is then displayed as an assembly of nodes, wherein only the nodesneeded for the particular view have been launched, and all other nodesare displayed without launching as part of another node or not at all.It is understood that various other embodiments will be apparent tothose of ordinary skill in the art, and that the invention is notlimited to the embodiments described herein.

1. A method of displaying visual content, said visual content comprisingat least two nodes, the method comprising displaying the visual contentassociated with at least a first node and at least a second node usingthe rendering method and coordinate system of the first node until apredetermined navigation condition is detected, and then displaying thesecond node using the rendering method and coordinate system of thesecond node.
 2. The method of claim 1 wherein at least one of saidrendering method or said coordinate system differs between said firstand second nodes.
 3. The method of claim 2 wherein said predeterminednavigation condition is a size of display of said second node.
 4. Amethod of displaying visual content organized into nodes, the methodcomprising displaying visual content associated with a first node usingits own coordinate system during a prescribed range of zooms anddisplaying visual content associated with said first node using acoordinate system of a second node outside said prescribed range ofzooms.
 5. The method of claim 4 wherein said first node is within saidsecond node.
 6. The method of claim 5 wherein said first node isdisplayed using its own rendering method during said range of zooms andusing a rendering method of said second node outside said range ofzooms.
 7. The method of claim 4 or 6 wherein at a border of said rangeof zooms, said visual content is displayed by blending from a firstrendition to a second rendition.
 8. The method of claim 7 wherein saidsecond node contains a pointer to said first node.
 9. (canceled)
 10. Themethod of claim 7 wherein said first rendition and second rendition usedifferent coordinate systems.
 11. The method of claim 7 wherein saidfirst rendition and said second rendition use different renderingmethods.
 12. The method of claim 7 wherein said first and secondrenditions use different rendering methods and different coordinatesystems.
 13. A method of displaying visual content, said methodcomprising displaying visual content associated with a first node usinga coordinate system and rendering method of a second node during a firstrange of zooms, displaying said visual content associated with saidfirst node using its own coordinate system and rendering method during asecond range of zooms, and ceasing to display said visual contentassociated with said first node during a third range of zooms.
 14. Themethod of claim 13 wherein said visual content gradually blends atborders of said first and second range of zooms, and said second andthird range of zooms. 15-22. (canceled)
 23. A method of substantiallysimultaneously displaying visual content associated with each of aplurality of nodes, each node having its own coordinate system, themethod comprising each node deriving information indicative of its ownposition in a display area, and displaying visual content associatedwith each of said nodes substantially simultaneously in said displayarea.
 24. The method of claim 23 wherein said deriving for at least onenode comprises receiving position information from another node, andconverting said position information from a first coordinate system to asecond coordinate system.
 25. The method of claim 24 wherein saidposition information is a position associated with a predeterminedlocation within said display area.
 26. The method of claim 25 whereinvisual content associated with each of said nodes is displayed using arendering method associated with the particular node.
 27. A method ofdisplaying visual content comprising displaying a first node having afirst coordinate system, displaying a second node within said coordinatesystem, and then displaying said second node in its own coordinatesystem in response to navigation by a user.
 28. The method of claim 27wherein said second coordinate system is more precise than said firstcoordinate system.
 29. The method of claim 28 wherein a precisionassociated with a coordinate system is measured in number of digits, andwherein said number of digits is variable.
 30. The method of claim 29wherein said precision associated varies up to a maximum limited bystorage capacity of a computer on which said method is operating.
 31. Amethod of displaying visual content on a display, the method comprisingdisplaying plural nodes at plural sizes, each in a specified coordinatesystem, each coordinate system having a precision, the precision inwhich a coordinate system is displayed being dependent upon the size atwhich the node is displayed.
 32. The method of claim 31 wherein saidprecision is variable and increases when the coordinate system in whicha node is displayed changes during navigation.
 33. The method of claim32 wherein said precision is variable and decreases when the coordinatesystem in which a node is displayed changes during navigation.
 34. Amethod of displaying nodes comprising displaying a node in a firstcoordinate system, and displaying the node in a second coordinate systemby gradually blending from said first coordinate system to said secondcoordinate system during a predefined range of navigation, wherein arendering method used for said node also gradually changes from a firstsuch rendering method to a second rendering method.
 35. (canceled) 36.The method of claim 35 wherein said navigation comprises either zoomingor panning.
 37. The method of claim 36 wherein information about a viewis propagated among plural nodes as said navigation occurs.
 38. Themethod of claim 37 wherein said propagation is accomplished using amathematical tree.